Niklaus Emil Wirth introduced the innovative concept of Programming = Algorithm + Data Structure [109]. Inspired by this, we advance the concept to the next level by stating that Design = Algorithm + Architecture. With a concurrent exploration of algorithm and architecture called algorithm/architecture co-exploration, this paper provides an overview of the leading paradigm shift in advanced visual and signal processing system design from embedded systems to the cloud and edge. As algorithms with high accuracy become exceedingly more complex and edge or Internet-of-Things generated data become increasingly larger, flexible parallel and reconfigurable processing are crucial in the design of lightweight systems with low complexity and low power. Therefore, the intelligent designs crossing levels of algorithm, system architecture, and microarchitecture, based on algorithmic-intrinsic complexity assessments, including efficient computation, data storage, data transfer, and potentials for parallelism are crucial and are surveyed. In particular, at the algorithmic level, this paper surveys state-of-the-art learned image and video codecs and their low-complexity implementations. The analytics architecture is also overviewed to explore the joint algorithmic and architecture co-design space. Furthermore, we survey intelligent technologies to control the system temperature and power consumption under a safe computing environment for low-power design at the microarchitecture level.